In a natural setting, speech signals are usually perceived against a background of many other sounds. The human ear has the uncanny ability to efficiently separate speech signals from a plethora of other auditory signals, even if the signals have similar overall frequency characteristics, and are coincident in time. However, it is very difficult to achieve similar results with automated means.
Most prior art methods use multiple microphones. This allows one to obtain sufficient information about the incoming speech signals to perform effective separation. Typically, no prior information about the speech signals is assumed, other than that the multiple signals that have been combined are statistically independent, or are uncorrelated with each other.
The problem is treated as one of blind source separation (BSS). BSS can be performed by techniques such as deconvolution, decorrelation, and independent component analysis (ICA). BSS works best when the number of microphones is at least as many as the number of signals.
A more challenging, and potentially far more interesting problem is that of separating signals from a single channel recording, i.e., when the multiple concurrent speakers and other sources of sound have been recorded by only a single microphone. Single channel signal separation attempts to extract a speech signal from a signal containing a mixture of audio signals. Most prior art methods are based on masking, where reliable components from the mixed signal spectrogram are inversed to obtain the speech signal. The mask is usually estimated in a binary fashion. This results in a hard mask.
Because the problem is inherently underspecified, prior knowledge, either of the physical nature, or the signal or statistical properties of the signals, is assumed. Computational auditory scene analysis (CASA) based solutions are based on the premise that human-like performance is achievable through processing that models the mechanisms of human perception, e.g., via signal representations that are based on models of the human auditory system, the grouping of related phenomena in the signal, and the ability of humans to comprehend speech even when several components of the signal have been removed.
In one signal-based method, basis functions are extracted from training instances of the signals. The basis functions are used to identify and separate the component signals of signal mixtures.
Another method uses a combination of detailed statistical models and Weiner filtering to separate the component speech signals in a mixture. The method is largely founded on the following assumptions. Any time-frequency component of a mixed recording is dominated by only one of the components of the independent signals. This assumption is sometimes called the log-max assumption. Perceptually acceptable signals for any speaker can be reconstructed from only a subset of the time-frequency components, suppressing others to a floor value.
The distributions of short-time Fourier transform (STFT) representations of signals from the individual speakers can be modeled by hidden Markov models (HMMs). Mixed signals can be modeled by factorial HMMs that combine the HMMs for the individual speakers. Speaker separation proceeds by first identifying the most likely combination of states to have generated each short-time spectral vector from the mixed signal. The means of the states are used to construct spectral masks that identify the time-frequency components that are estimated as belonging to each of the speakers. The time-frequency components identified by the masks are used to reconstruct the separated signals.
The above technique has been extended by modeling narrow and wide-band spectral representations separately for the speakers. The overall statistical model for each speaker is thus a factorial HMM that combines the two spectral representations. The mixed speech signal is further augmented by visual features representing the speakers' lip and facial movements. Reconstruction is performed by estimating a target spectrum for the individual speakers from the factorial HMM apparatus, estimating a Weiner filter that suppresses undesired time-frequency components in the mixed signal, and reconstructing the signal from the remaining spectral components.
The signals can also be decomposed into multiple frequency bands. In this case, the overall distribution for any speaker is a coupled HMM in which each spectral band is separately modeled, but the permitted trajectories for each spectral band are governed by all spectral bands. The statistical model for the mixed signal is a larger factorial HMM derived from the coupled HMMs for the individual speakers. Speaker separation is performed using the re-filtering technique.
All of the above methods make simplifying approximations, e.g., utilizing the log-max assumption to describe the relationship of the log power spectrum of the mixed signal to that of the component signals. In conjunction with the log-max assumption, it is assumed that the distribution of the log of the maximum of two log-normal random variables is well defined by a normal distribution whose mean is simply the largest of the means of the component random variables. In addition, only the most likely combination of states from the HMMs for the individual speakers is used to identify the spectral masks for the speakers.
If the power spectrum of the mixed signal is modeled as the sum of the power spectra of the component signals, the distribution of the sum of log-normal random variables is approximated as a log-normal distribution whose moments are derived as combinations of the statistical moments of the component random variables.
In all of these techniques, speaker separation is achieved by suppressing time-frequency components that are estimated as not representing the speaker, and reconstructing signals from only the remaining time-frequency components.